A Comparison of Item and Testlet Selection Procedures

Testlet response theory (TRT) is a measurement model that can capture local dependency in testlet-based tests. One of the purported advantages of TRT over the more commonly-used polytomous IRT approach to modeling testlet-based tests is that it allows for ad hoc testlet construction in a testlet-based computer adaptive test (CAT). The goal of this study was to investigate the merits of such a CAT design. Specifically, it examined the use of testlet-based CATs that not only chose each testlet adaptively, but they also adaptively selected each item within the testlet, based on the estimated examinee proficiency. This design was termed a CAT with adaptive testlets, and it was compared against a CAT whose within-testlet items were all pre-determined and fixed (termed a CAT with fixed testlets). Real data from a large-scale assessment were calibrated using the 3PL-TRT model and used in this simulation study, which compared these testlet-based CAT designs on their measurement and exposure control properties. The study found that the use of adaptive testlets improved measurement precision while achieving better pool utilization rates. The use of an item-level exposure control procedure within a CAT with adaptive testlets resulted in similar measurement precision, but only a modest gain in pool utilization rates, when compared to a CAT with adaptive testlets and no item-level exposure control. This study represented an initial examination of the properties of CATs with adaptive testlets. As such, suggestions for future research are also provided.